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Abstract. In this work is considered the problem of the emergence of conventions in a 
society through distributed adaptation by agents from their on-line experiences at run 
time. The agents are connected to each other within a fixed network topology and interact 
over time only with their neighbors in the network. Agents recognize a social situation 
involving two agents that must choose one available action from multiple ones. The study 
confirms the emergence of system-wide conventions via the process of social learning 
where an agent learns to choose one of several available behaviors by interacting 
repeatedly with randomly chosen neighbors without considering the identity of the 
interacting agent in any particular interaction. 
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1. Introduction 


Favored by the large support of mass-media and of Internet, new ways for 
interaction and collaboration between individuals has arrived in the form of social 
networking. 


The tendency to group a growing number of entities (in particular individuals) is 
typical for complex systems, and this it done not by cluster size, but by the huge 
number of mutual interactions between members. 


Recent (let say the last twenty years) specialty literature indicated that networks 
are the best fit for this structure, and among these Scale-free Networks (SFN) 
having both topology and dynamic behavior (expressed by the flow of 
information) independent of scale representation seem to capture best the 
behavioral characteristics 


(Internet remains the best example, with its fractal topology and self-similar 
patterns exhibited in the informational traffic). 


In the same time, one can affirm that the main characteristic of complex systems 
with SFN structure is self-organization. 


Self-organization is the way due to which following rules (usually simple) of 
behavior emerges a new organizational structure and new interactions entities also 
appear in the system. 
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The term self-organization is used to describe the dynamics of complex systems 
and related emergent processes. Most often the interactions are nonlinear and the 
evolution is nondeterministic (somehow chaotic), but may lead to a new 
hierarchical order in the system. 


In this way, emergence is a step in the evolution of complex systems allowing 
their adaptation to changes in the environment and can also bring new ordering 
relationships, for example by limiting the degrees of freedom of movement. 


Through its relationships with social entities, an organization overcomes the 
entities' cognitive, physical, temporal, formal and informal barriers and thusly, 
through its formalism of group affiliation, its procedures, objectives and 
restrictions imposed to its members, it persists despite the fact that, alone or at 
group level the agents (i.e., the members of a social entity or of an organization) 
do not interact. 


Consequently, the organizations enjoy the advantages of its own purely formal 
structures which can exist independently of the individual objectives, actions and 
intentions of the agents, which it actually coerces to act in a certain manner, such 
as through objective differentiation and therefore though material benefits 
differences, with the objective of reaching the organization's goals. 


Conversely, an organization's formalism can become its disadvantage due to the 
inherent rigidity of the roles and structures of its makeup, which every so often 
are slow to react to open, dynamic and unpredictable environments. 


In order to compensate this disadvantage, the organization must employ complex 
social interaction capacities, such as cooperation and negotiation, as well as 
adaptability and efficiency in overcoming the problems of its members (e.g., loss 
of interest, lack of interaction etc.). 


Based on the above, we can conclude that organizations (i.e., systems composed 
of multiple agents/individuals) exist, react and persist due the common objectives 
and simultaneous actions of its members, and cannot be modeled like the social 
systems where, sociologically, the change of the systems' structure is due to the 
interaction between its members but not due to their intentions, which can be 
different. 


Taking into account the needs of those organizations to respond quickly to the 
specific challenges of a hyper connected world, in this paper we study the 
emergence of conventions in a society of artificial agents through the repeated 
interactions between its members. 


An on-line (or run-time) convention means that agents can tailor their decision to 
the current environment. Each agent learns from its interactions concurrently and 
over repeated interactions with randomly selected neighbors. 
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Hence, it is important to study mechanisms that lead to emergence of a convention 
from on-line interaction experience. 


Two particular approaches have been proposed to determine the development 
framework of this model: multi-agent system organization to observe how the 
members of such a system relate (for explanation of some elements of social 
emergence, such as collaboration, consensus, confidence) and respectively scale- 
free networks for evidence of some elements of organizational emergence (growth 
mechanisms and degree of interconnectivity). 


2. Related works 


The emergence of social conventions in multi-agent systems has been rigorously 
analyzed first in the seminal work of Delgado [1], with reference to settings where 
every agent may interact either with every other agent or with nearest neighbors, 
according to some regular underlying topology. 


These networks, one of the main examples being the Internet, are what is called 
complex, that is, either graphs with the small-world property or scale-free graphs. 
In this note is studied the efficiency of the emergence of social conventions in 
complex networks, with the conclusions that scale-free graphs make the system 
much more efficient than regular graphs with the same average number of links 
per node. 


The problem of cooperation in multi-agent systems was largely discussed in [2]. 
The novelty consists in proposing the use of machine learning techniques to 
automate the search and optimization process. 


Additionally, the authors discuss direct and indirect communication between 
agents in connection with learning, plus open issues in task decomposition, 
scalability, and adaptive dynamics. 


Successive progress in resolving the issue of emergence has been brought about 
by the work of a group of authors ([3], [4], [5], [6]) that have started from the 
definition of convention by reference to social norms and have studied different 
network structures, in order to compare and evaluate the effects of different 
network topologies on the success and rate of emergence of social conventions. 


Additionally they proposed a reward metric that takes into consideration the past 
action choices of the interacting agents, based on classical game theory. 


Finally, we mention some more recent developments ([7], [8], [9]) that 
simultaneously address the three trends highlighted in social systems research: 
machine learning algorithms for evaluating cooperation mechanisms in multi-agent 
systems, evolutionary games to determine the effectiveness of social interactions 
and accuracy of social networks models based on free-scale topologies. 
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Their contributions explain the emergence of cooperation, model the co-evolution 
of multiple network event streams to allow for more complex data structures with 
many types of nodes and events and introduce network statistics to reflect the 
potentially complex dependencies among agents. 


3. Basic principles of the General Theory of Information 
Definition of social interactions 


Let consider a population of N interacting agents. Although in the general case a 
convention could involve an interaction between many agents at the same time, in 
our study we discuss only the case of bilateral interactions, the most common in 
practice. For our definition of a bilateral social interaction, we consider (similar to 
the definition given in [6]) that each agent plays a specific role: first role (or row 
role) and second (or column) role, where the rows and columns are elements of a 
matrix (the payoff matrix) reflecting the preferences between the different 
outcomes of the interaction for each agent like in a normal-form game. 


Of course, we consider that each agent can experience both roles. The agent has to 
select an action in the set A, of actions available to the row role, respectively in the 
set A. of actions available to the column role). Each agent models the situation 
with a payoff matrix P; of size | A; | * | Ac |. 


Definition 1 (Social interaction) A social bilateral interaction is a 4-tuple 
{N,A,Ao(Pi)ien} where: 


— agent i gets P; (a,a,) when i is the row agent and chooses action a, and 
the other agent is the column agent that chooses a,. 


— agent i gets P; (a,,a,) when i is the column agent and chooses action a, 
and the other agent is the row agent that chooses a, . 


We assume that all agents have a similar understanding of the social situation: all 
the agents share the same ordinal preference over the different outcomes of the 
game, but they may have different cardinal preferences. This assumption excludes 
from consideration situations where agents that have different preferences 
between the outcomes. 


By the other hand, we allow the actual payoffs to differ from agent to agent, and 
we allow indifference: if some agents strictly prefer outcome a over outcome b, 
then no agent strictly prefer b over a, but some may be indifferent over them. 


Topologies of social networks 


Agents are connected by a interconnection graph G of a fixed topology that 
restricts their interactions only with their direct neighbors (we represent agent 
relationships with nodes and links). 
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Every node i represents an agent (actor) i within the network and links (i, 7) denote 
social ties between agents i and j. 


More of that, we decorate each link (i, 7) with the strength of the social tie or the 
amount of information flowing through it, hereafter called link weight w;,. 


The statistical analysis of link weights w;; between pairs of vertices in the social 
network indicates an heterogeneous pattern of interactions, typically following a 
power law: P(wij)~wij” where P(w;;) is the probability of having a link with 
weight Wij. 


In addition, the heterogeneous distribution of link weights might be related to the 
hierarchical organization of the social network. 

In this aim, we define P(w, yw. 7 as the cumulative distribution function (cdf) 
of P(w, ,)and Pw, j)=1—P(w,,,) as the complementary cumulative distribution 
function (ccdf), or simply the tail distribution. 


For example, in fig. 1 are represented the tails for the distribution of the number 
of agents (friends) per user in a simulation of a small social network with around 
500 users, were each user is represented as a node in a graph. Each node is 
encoded with an integer. 


The analysis of the graph was made for two cases, depending of the social relation 
between two users A and B, which can be asymmetric or symmetric. 


An asymmetric relation (fig. 1 a) means that if node A has an edge to node B, this 
does not mean that B also has an edge to A. 


Of course, in the symmetric relation (fig. 1 b) there is a bidirectional link between 
A and B. 


5 T T T 
number of friends per user + ; number of friends per user (symmetric graph) at 


ccdf log 
ccdf log 


number of friends (log) 


number of friends (log) 


a. b. 
Fig. 1. ccdf of the friendship graph: a) asymmetric; b) symmetric. 
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Both distributions seem to be heavy tailed, which means that the properties of the 
graph do not change if the edges in the graph are directed or undirected. 


For these reason in all other simulations we have considered only the symmetric 
version of the graph, which is more appropriate with the real case of open social 
networks (OSS). 


There is a characteristic pattern of symmetric interaction, where a few strong units 
dominate the activity of the whole OSS. Interestingly, the distribution of link 
weights in large software communities also follows a power-law; with an 
exponent consistent with the observed in the small software communities. 


Most real networks typically contain parts in which the nodes (units) are more 
highly connected to each other than to the rest of the network. The sets of such 
nodes are usually called clusters, communities, cohesive groups, or modules 
having no widely accepted, unique definition. 


This remark leaded us to consider that the most suitable topology of a social 
network is that of a scale free network [10]. 


Fig. 2 illustrates the topology of a free scale network with 128 nodes that started 
from an initial core of 4 nodes; in the connection of other nodes we have applied 
the law of the preferential attachment. 


Fig. 2. A network with scale-free topology. 


Social interaction protocol 


To resume, we consider in our model a population of N of n = |N| agents located in 
a graph G that faces a social situation involving two roles, with their 
corresponding action set A,, A,, and each agent i models the social situation with a 
game having a payoff matrix Pi. Hence, we represent the social situation as a 
tuple {N,G,A,AoPi, ...,Pn}. 
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The simulation of the system progresses in discrete steps. At each iteration, many 
distinct pairs of agent are randomly generated. To make a pair, an agent is first 
chosen from the set of agents that has not already been selected, and is paired with 
a randomly selected neighbor that has not yet been selected. This selection 
process is iterated until no more pairs can be formed, i.e., when there are no more 
neighbor agents that have not been selected. 


In the case of a fully connected graph, this algorithm will produce N/2 pairs at 
each iteration, and all the agents will learn at the same speed. 


However, when agents do not have the same number of connections, the agents 
that have more neighbors are more likely to experience an interaction at each 
iteration. 


This selection imbalance introduces a bias: agents with more connections will 
accumulate more experience and will learn faster than agents with a small number 
of connections. 


Because of the topology of the graph, agents with many connections will also 
have a greater influence on others. 


4. Theoretical approaches in the Information Science 


Definition of a social convention 


The behavior of an agent in repeated play of a bilateral stage game is 
characterized by its actions when it plays respectively the row role and the column 
role. 


More formally, given a social interaction {N,G,A,,A,P1, ...,Pn}, the behavior of 
an agent i is a pair (r', c' ) that consists of a pure strategy r' € A, for the row role 
and a pure strategy c' © A, for the column role (the strategies are the actions 
available to the agents). A convention corresponds to an equilibrium strategy 
profile for all pairs of agents in the population. It is a simplified form of the 
consensus principle used in Decision Support Systems (DSS) philosophy. 


Definition 2 (Social convention) For a social situation {N,G,A,,AcP1, ...,Pn}, we 
say that the population uses a convention when for all pairs of agents (i, j ), we 
have both that (r’, c’) is an equilibrium for the game (P;, P’;) and ((r’, c') is an 
equilibrium for the game (( Pi P;)where P’ is the transpose of P. 


Not all equilibria are conventions, and in the following, we will consider that a 
convention is a pure strategy Nash equilibrium. The use of a Nash equilibrium 
ensures that the equilibrium is stable: knowing that other agents are following the 
convention, a given agent is incentivized to follow it as well. 


We consider only pure strategies as, in practice, conventions are pure strategies. 
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In Nash equilibrium the two players adopt a pair of strategies such that neither 
player can get a better payoff by deviating from their strategy. 


In other words, each strategy is a best response to the other. Depending on the 
game, there may be no Nash equilibrium, a unique one, or much equilibrium. 
Because in these strategies the players are not allowed to use randomness to 
decide their moves, they are called pure strategies. 


Now is the moment when Machine Learning (ML) as technique Artificial 
Intelligence (AI) of meets Game Theory. Many situations involving many agents 
can be modeled as games where learning algorithms can then be used as decision 
making mechanisms. As a result, learning to play repeated games has been an 
active area of research in multiagent systems and in particular in social networks, 
when considering that all agents would adopt the same learning algorithm in a 
decentralized environment. 


Machine learning algorithm 


We assumed that the basic decision making mechanism of an agent is a learning 
mechanism. This allows the agents to adapt their behaviors to their current 
environment and reduces the need for the system designer to specify precise 
parameters for each environment. 


We will further assume that agents try to learn a behavior at the level of a social 
interaction. For example, an agent does not try to learn a behavior that depends on 
the specific agent he interacts with. If an agent fails to learn an appropriate 
behavior at this level, we would assume that the agent would refine her model of 
the situation (e.g. consider different sub-situations or learn some exceptions). 
Then, the agent would be able to recognize a given situation and use the 
appropriate data to make a decision. 


With the social learning framework, there is no theoretical guarantee that a 
convention does emerge and stabilize. The goal of the agents is not to learn or 
discover a convention. Rather, their goal, as rational agents, is to choose decisions 
to maximize their expected utility. In order to test whether it was possible for 
learners to choose a correct behavior, we chose reinforcement learning as our tool 
for decision making. 


In this aim we are studying the emergence of a convention in a population of 
interconnected interacting agents. Each agent uses a learning algorithm to learn, 
from accumulated experience, how to behave in each role of the social situation. 
We will at first assume that the agents do not have any initial bias towards a 
particular equilibrium. 


We want to observe whether the population is able to learn the same behavior, 1.e., 
whether, in the long run, the population adopts a convention. 
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To learn a useful behavior, an agent needs to first explore its options and 
subsequently exploit its accumulated knowledge. Even when the behavior of other 
agents appears predictable, an agent will need to explore periodically to ensure 
that it is not using a sub-optimal strategy. 


It is also important in case of a change in the environment or if they change 
environments. This is particularly important for open and dynamic agent societies 
that are of interest to us and to a large percentage of researchers in the multi-agent 
systems community. 


Consequently, to identify the emergence of a convention, we adopted a common 
definition from literature [11]. 


Definition 3 (Convention emergence) For a social {N,G,A,AoPi, . ~~. ,Pn}, a 
convention has emerged when the strategy profile (r,c) <A;xA, is played by T % 
of the population in a given iteration. The threshold T is different in various 
contributions, but all the authors agree that it should be at least 90%. 


Note that each agent must learn how to behave for each role of the social 
situation. In this paper, we assumed that both roles are learnt independently. Of 
course, this issue is relevant for interactions that are not symmetrical. For 
symmetric problem, we would simply need a single learning algorithm. 


Now we consider agents situated in more restrictive interaction topologies. Each 
agent is represented by a node in the network and the links represent the possibility of 
interaction between nodes (or agents). We consider that agents form a one- 
dimensional lattice with connections between all neighboring vertex pairs. We can 
see that when increasing the neighborhood size, the convergence time is steadily 
reduced. This effect is due to the topology of the network. When the one dimensional 
lattice has a small neighborhood size, on average, the diameter of the graph is high 
and therefore agents located in different parts of the network need a higher number of 
interactions to communicate their decisions or arrive at a consensus. 


When agents have a small neighborhood size, they will interact often with their 
neighbors, resulting in diverse sub conventions forming at different regions of the 
network. We note that in each interaction, both agents are learning from it, therefore 
agents reinforce each other in each interaction. Such divergent sub conventions 
conflict in overlapping regions. To resolve these conflicts, more interactions are 
needed between agents in the overlap area between regions adopting conflicting sub 
conventions. 


Unfortunately, the agents in the overlapping regions may have more connections in 
their own sub convention region and hence will be reinforced more often by their 
sub conventions, which makes it harder to break sub conventions to arrive at a 
consistent, uniform convention over the entire society. 
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On the other hand, when neighborhood sizes are large, and hence network 
diameters are small, agents interact with a large portion of the population. As a 
result, it is less likely that sub conventions are created or sustained. 


Emergence of conventions in scale-free networks 


We observe an interesting phenomenon for scale-free networks: sub-conventions 
might be persistent and the entire population fails to converge to a single 
convention. This phenomenon can be explained by some particular structure of 
the network. 


In general, each node i of a network can be characterized by a membership 
number mj, which is the number of communities the node belongs to. In turn, any 


two communities a and # can share s%", nodes, which we define as the overlap 


size between these communities. 


Naturally, the communities also constitute a network with the overlaps being their 
links. The number of such links of community a can be called as its community 


com 
degree, d<”". 


Finally, the size of any community a can most naturally be defined as the number 
of its nodes. To characterize the community structure of a large network we 
introduce the distributions of these four basic quantities. In particular, we will 
focus on their cumulative distribution functions denoted by P(s°”), P(d°”), 
P(s°"), and P(m), respectively. 


The basic observation on which our definition for community relies is that a 
typical community consists of several complete (fully connected) subgraphs that 
tend to share many of their nodes. 


Thus, we define a community, or more precisely, a k-clique-community (the term 
was introduced in [12]) as a union of all k-cliques (complete subgraphs of size k) 
that can be reached from each other through a series of adjacent k-cliques (where 
adjacency means sharing k—1 nodes). 


This definition is aimed at representing the fact that it is an essential feature of a 
community that its members can be reached through well connected subsets of 
nodes. 


There are other parts of the whole network that are not reachable from a particular 
k-clique, but they potentially contain further k-clique-communities. 


In turn, a single node can belong to several communities. All these can be 
explored systematically and can result in a large number of overlapping 
communities. Notice that in most cases relaxing this definition (e.g., by allowing 
incomplete k-cliques) is practically equivalent to lowering the value of k. 
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In the same time any k-clique (complete subgraph of size k) can be reached only 
from the k-cliques of the same community through a series of adjacent k-cliques 
(two k-cliques are adjacent if they share A—/ nodes). The algorithm for numerical 
determination of the full set of k-clique-communities is based on first locating all 
cliques (maximal complete subgraphs) of the network and then identifying the 
communities by carrying out a standard component analysis of the clique-clique 
overlap matrix [8]. We use our method for binary networks (i.e., with undirected 
and unweighted links). 


An arbitrary network can always be transformed into a binary one by ignoring any 
directionality in the links and keeping only those that are stronger than a threshold 
weight w*. Changing the threshold is like changing the resolution with which the 
community structure is investigated: by increasing w* the communities start to 
shrink and fall apart. A very similar effect can be observed by changing the value 
of k as well: increasing k makes the communities smaller and more disintegrated, 
but at the same time, also more cohesive. 


The extent to which different communities overlap is also a relevant property of a 
network. Although the range of overlap sizes is limited, the behavior of the 
cumulative overlap size distribution P(s°’) is close to a power law for each 
network, with a rather large exponent. 


5. Conclusions 


We investigated a bottom-up process for the emergence of social convention that 
depends exclusively on individual experiences rather than observations or 
hearsay. The proposed social learning framework requires each agent to learn 
from repeated interactions for a given social situation, without using knowledge of 
the identity of the other agents involved in the interactions. 


The goal of this work was to evaluate whether such social learning can 
successfully evolve and sustain a useful social convention that resolves conflicts 
and facilitates coordination between population members. 


The experimental results confirm that such distributed, individual, and social 
learning is indeed a robust mechanism for evolving stable social conventions. The 
results also suggest that to deploy a multiagent system, one can use generic agents 
that use learning mechanisms which can, with no detailed knowledge of the 
environment, learn efficient and stable coordination behavior. 


Additionally the results confirm that stable conventions arise in scale-free 
networks because of some inherent structural characteristics of these networks. 
For the future we plan to investigate, in further depth, the reasons why these 
conventions might be created and maintained, as well as, mechanisms to dissolve 
them. 
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